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ABSTRACT 

This manual was written for L ^th environmental 
educators who want to evaluate the environmental education (EE) 
instructional materials they use or recommend to teachers, and 
inservice providers who want to help teachers evaluate EE materials 
in workshop settings. This information is provided to expand 
knowledge and expertise, facilitating informal yet thorough 
evaluations, and to help educators instruct others to do the same. 
The user is guided through an evaluation process and provided with 
tips, suggestions, criteria, worksheets, and workshop activities. 
Sections cover evaluation basics, evaluation criteria, developing an 
evaluation strategy, formal evaluation projects, workshop activities, 
and activity masters. A resources section contains a brief glossary 
of evaluation terms and 20 references to evaluated EE materials, 
evaluation tools, and additional readings. The manual is not intended 
to direct the development of formal evaluation projects that require 
statistically significant results. It is a practical guide, one 
designed to help users simply but effectively examine EE 
instructional materials to determine whether they meet their needs. 
(LZ) 
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Sieve and Anita Take Stock 



"I was nervous as we planned this workshop but the teachers really seemed to learn a lot!" 
exclaimed Steve, a resource specialist. He and Anita, a district environmental 
education coordinator, were meeting to discuss their recent biodiversity workshop. 

"They seemed to have a good time too, but let's see how they evaluated us," Anita 

proposed as she began looking through the surveys teachers Hlled out at the end 
of the workshop. Anita pointed to a few of the completed evaluations, noting, 
"Several teachers did say they were overwhehned with the volume of materials." 

Looking over her shoulder at the evaluation forms, Steve said, "I wonder if we might 

be wasting some trees. Can teachers really use all those materials? Some of them 
took so much stuff, they had to make two trips to their car!" 

"I remember when I was a teacher," Anita recalled. "Most of the resources I got at work- 
shops turned out to be dust colleaors. I just didn't have time to sort through all 
that stuff to figure out which were worth using. I still have that problem!" 

She and Steve decided it would be better to provide teachers with only the best resources. 
They agreed it made sense to evaluate the resources they provide, just as they 
always evaluate the success of their workshops. 

"It seems like a big task," said Steve. "I've read some articles in the Journal of Environ- 
mental Education (JEE), and all of them stressed being very systematic about 
evaluations, developing a rating scale, and setting up a committee of teachers. 
It seems like we should get someone at the university to do it." 

"I don't think we need to be that technical about it for our own personal use," replied 
Anita. "I'm sure we can combine our knowledge of biodiversity and education 
with some of the information from those JEE articles and come up with an 
evaluation strategy that works for us and our workshop participants." 

"I don't know, Anita. Do you think we'll be able to predict teachers' needs?" Steve asked. 

Anita thought for a moment. "I have an idea," she said. "What if we build some time 
into our workshop for teachen^ to learn about a couple of evaluation scales. 
Then, using those scales as examples, they can selea or create their own criteria, 
scales, and evaluation methods." 

"Great idea!" Steve added. "We can figure it out and then pass those skills on to teachers 
and they can improve on our work. That way they can apply evaluation to all 
their EE materials." Anita and Steve agreed to do some research on their own. 
They planned to meet again to evaluate their materials and begin developing 
a workshop session on evaluation. 
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ONE OF THE REASONS TEACHERS ATTEND INSERVICE PROGRAMS is tO 
find new instructional resources. As inservice providers, we respond 
to this need by distributing instructional materials and handouts. 
Walk into any environmental education ( EE) teacher workshop and 
you will see the walls covered with posters and tables piled high with 
supplemental resources containing content background, activities, and new 
teaching tips. EE conferences have rooms full of exhibits displaying new activity 
guides and other instructional materials. 

Often teachers have no trouble finding EE materials, rather for 
many it is a daunting task to choose from all that are available. Teachers often 
have neither the time nor the confidence to sort through and choose the best 
instructional materials for incorporating EE into their curriculum. You can help 
teachers increase the quality of their lessons by providing simple and effective 
ways to evaluate EE instructional materials. Furthermore, evaluation can 
familiarize teachers with environmental education concepts. Teachers can 
make better choices to meet any numbv-i of objectives and individual needs, 
such as: 

- meet state and local learning objectives 

- address environmental education goals 

- convey accurate information 

- include relevant context for various socio-economic gi'oups 

- relate age-appropriate concepts 

- adapt to different teaching styles 

- accommodate the different learning styles of their students 

- select interesting and easy-to-use materials 

- fit individual constraints, such as time or money. 

All educators can gain the skills and knowledge to evaluate environmental 
education materials. We will provide you with background information, 
simple tools, relevant examples, and activities. You can use these to evaluate the 
materials you present in workshops and to lead activities that build teachers' 
evaluation skills. 
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This unit focuses on evaluation ot written environmental education 
instructional material, most specifically activity guides. However, the criteria 
and techniques discussed here can be adapted for use with many types of 
instructional material such as video, softv.'are, and texts. 

The sections in this unit focus on the following steps: 



l.t'Liltuition Basics 

Background information puts the evaluation 
process in a larger context, helping you to 
better understand evaluation procedures, 
!;uidelines, and criteria. Included in Section 
1 are basic terminology and key concepts. 
A more complete glossary of terms is pro- 
vided on page 44. 

i'.valtiation Criteria 

The heart of any evaluation effort is the 
criteria used to rate the instructional mate- 
rials. There is no single answer to what 
makes a good activit)' guide. However, there 
are several aspects that every EE activit)' 
guide should include. Section 2 provides 
a framework for developing criteria and 
suggests specific criteria relevant to EE. 
I ise this information to generate a list 
c f criteria that meets your specific needs. 

Det'elopiitji Your Evaluation .SYrarei^' 
Just as no single set of criteria will meet 
all evaluation needs, no one evaluation 
strategy applies o all EE activity guides. 
Selecting a strategy can be a confusing 
task. This section (pages 11-15) demysti- 
fies evaluation strategies by providing both 
guidelines for modifying existing tools and 
step-by-step procedures for developing 
your own. 

A Look at Formal Evaluation Projects 
just because an activit)' guide has been for- 
mally evaluated and approved by a partic- 
ular organization does not guarantee it 
will satisfy your needs. The guidelines and 
real-life examples in Section 4 can help 
you evaluate methodologies. You can also 
use this section to understand some of 
vour options if you arc interested in devel- 
oping a more formal evaluation project. 




Most instructional materials 
may be useful even if they do 
not rate well on ail criteria. It 
is best to focus on strengths, 
weaknesses, and usefulness 
in your evaluations. 



What To Do in a Workshop 
Teachers are more likely to evaluate materi- 
als if they receive hands-on experience in 
your workshops. Section 5 provides ideas, 
tips, and sample agendas to help you incor- 
porate evaluation topics into your work- 
shops. Several strategies are provided to 
help you choose an approach that will 
work best in your workshops. 

Workshop Activities and Sample Tools 
Section 6 complements the previous 
section by matching the workshop ideas 
with specific activities. Each activity is 
designed so you can use or modify it for 
your particular workshop. 

Resources 

This document may leave you wanting 
more information, ideas, and details. The 
Resource Section provides a bibliography 
of evaluation projects and additional 
readings. 
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Evaluation Basics 

IKE ANY OTHER DISCIPLINE, evaluation has unique terms and con- 
I cepts. This section presents some of the basic tenets of evaluation and 
I environmental education. Understanding this background will help 
^ m J prepare you to adapt evaluation criteria and methods (described in 
sections II and III) without compromising effectiveness. 



A Few Terms 

Terms such as "curriculum," "instructional 
material," and "aaivity guide" are often 
used interchangeably. Perhaps they have 
very specific meanings for educators in 
your area. To avoid confusion in this docu- 
ment, we use these terms very specifically. 

A curriculum is a complete course of study 
carried out over the semester or entire school 
year to meet a specific set of learning objec- 
tives. The learning objectives may be estab- 
lished by teachers, school districts, state 
guidelines, or national standards. 

Instructional materials are resources that 
teachers use to address the curriculum. 
Such resources may include texts, activity 
guides, videos, and other materials designed 
specifically for classroom use, as well as 
newspaper articles, movies, comic books, 
and other sources of information on envi- 
ronmental concepts or issues. 

Actiinty guides are one type of written 
instructional material that often contain 
background information, student activities, 
and suggested lesson plans. Many activity 
guides cover specific learning objeaives 
and are designed as supplements, form a 
unit, or make up a course. 

Evaluation of instructional materials is 
the systematic application of criteria to 
judge whether the materials will meet the 
user's goals. 



These distinctions arc intended to make one 
point very clear: most instructional mate- 
rials should not be evaluated as if 
they will make up an entire curriculum. 
As part of a larger picture, they may be 
useful even if they do not rate well on all 
criteria. Hence, this unit focuses on identi- 
fying the strengths, weakness, and useful- 
ness of written EE instructional materials 
for the specific needs of teachers. 

Another set of terms is used to describe 
methodology. To help you meet your 
goals of developing a simple and effective 
strategy, the following sections outline 
an informal approach with the following 
characteristics: 

Amicijair Reinew: Emphasis is on an 
indirect evaluation of materials (before 
you use mem) to select those that best meet 
your needs. 

Teacher usefulness: These strategies 
include criteria based on goals of EE and 
how well the materials meet specific needs 
of teachers (vs. the goals of the authors). 

Limited scope: The criteria and strategies 
in this unit are best suited to evaluate 
EE activity guides. Some criteria may not 
be appropriate if you are evaluating other 
types of instructional resources. 

single or Comparative: Criteria can 
be applied to one activity guide or used 
to compare several guides. 
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A Few Key Concepts 



You may have an intuitive sense of what 
works, or know which t>-pe ot activities are 
usually popular with students. Yet, if asked 
to provide an "'evaluation," you may not 
know how to support your intuition. 
Is it enough to know whether activities 
excite students? Is helping them learn one 
or two concepts sufficient? You may find 
it difficult to define all of the factors that 
make an activity work with students. 

One way to evaluate EE activity guides 
is to think about your goals as a teacher 
educator. Your role is to motivate teachers 
and help them develop a good knowledge 
hase, skills, and a desire to bring HE to 
their classrooms. In measuring the success 
of your workshop, you may gauge whether 
they became more motivated, knowledge- 
able, skilled, and active environmental 
educators. Likewise, evaluating EE activity 
guides considers many of the same types 
of elements: reactions, learning, behavior, 
and results.' 

lifLiction I'.viduatiim measures the users' 
immediate positive or negative response 
to the learning experience. This is the 
most common level of evaluation. An arm- 
chair review may predict how students will 
respond to the material. 

Kttowled^ic Evaluation measures whether 
using the activities with students increases 
iheir knowledge and awareness of environ- 
mental issues and influences attitudes. 
In an armchair evaluation you can assess 
the quality of the content and if teaching 
methods meet your students needs. 



Bclhirior iivahiatiim measures whether 
tlie learning experience alters student 
behaviors. Your armchair review could 
determine whether the materials provide 
the knowledge and skills the learners need 
to be able to engage in new behaviors. 

Results liraluatioii measures whether the 
behavioral changes have an overall impact 
on the environment or students' lives. 

It is very hard to measure whether an impact 
on students' lives is due to the instructional 
materials or any of a million other poten- 
tial factors. In fact, evaluating for results is 
outside the scope of most informal evalua- 
tions and will not be discussed in this unit. 
Let's explore how you may achieve reac- 
tion, knowledge, and behavior evaluation 
in an armchair setting. 



Comparing Direct 
vs. Indirect Evaluation 

A direct evaluation of EE activity guides is 
easy to envision. Teachers can gauge student 
responst and use a variety of assessment 
tools to find out whether students learned. 
In the context of an informal armchair 
review, you can only gauge your responses 
to the material. However, you can still add 
depth to your evaluations by focusing on 
the likelihood that the activity guide will 
elicit a positive response, help students 
learn, and change behaviors. 

Direct evaluation usually yields a stron- 
ger certainty, but it is difficult to evaluate 
more than a few activities. With an indirect 
review you can critique many activity guides 
in one sitting. The two scenarios in the 
box on the next page illustrate how you 
might indirectly predict which guides will 
be useful. 
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Examples of Direct Evaluatkm and Indirect Review 

Layton and Miguel have each received a new activity guide on recycling. 
Layton, a high school teacher, decides to evaluate it by using it with his 
class. Miguel, a resource specialist with a local governmental agency, does 
an armchair evaluation before using it in his next workshop. 



Layton's Direct Evaluation 

Reaction: Layton noted students' 
general level of enthusiasm, atten- 
tion span, how many questions 
students asked and how hard it was 
to encourage discussion. 

Knowledge: Layton started his c':.- 
session by having students write a 
short essay about the impact of solid 
waste in their lives. He compared 
the level of knowledge in the essays 
to test scores from a post-lesson quiz. 

Behavior: Students asked permission 
to set up a classroom recycling project. 
Layton noted whether the unit pro- 
vided them with the motivation and 
skills to accomplish this task. 



Miguel's Indirect Evaluation 

Reaction: Miguel listed the attributes 
of fun learning expariences. He 
reviewed the activity guide for such 
factors as providing a variety of 
learning situations, relevancy to 
students in his area, and hands-on 
activities. 

Knowledge: Miguel reviewed the 
activity guide to determine if key 
concepts were present, how well 
they were addressed from a variety 
of perspectives, and whether prob- 
lems and solutions were presented 
so students may identify appropri- 
ate roles for themselves. 

Behavior: Miguel looked to see 
whether activities presented infor- 
mation on a broad range of feasible 
action strategies and included activ- 
ities directed at building skills stu- 
dents would need to engage in such 
actions. 
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Evaluation Criteria 



CRITERIA ARE THE HEART of any evaluation effort. Now that you have 
a sense of what an armchair evaluation is, it's time to start defining 
the specifics. The number and type of criteria you use and the methods 
used to apply rhem will all effect your results. These specifics make 
each evaluation effort unique. 

There are different categories of criteria you may consider for your 
evaluation. Educators tend to focus on the quality and accuracy of content. 
Learning styles and presentation of the material are also important consider- 
ations. Many EE evaluation effots measure against the goals of environmental 
education Often teachers use state or local education objectives to define the 
content goals. This section provides a framework for developing criteria based 
on three overlapping aspects: environmental education goals, pedagogy, and 
materials design. Together, they can create a well-rounded, thoughtful evalu- 
ation. After you identify' relevant criteria. Section III wiii help you place them 
in a usable format. 

As you read this section, remember that the focus is on EE activity 
guides; there will be categories or specific criteria that do not apply to other 
types of instructional materials. For example, you would not judge a text on 
classification of trees against criteria for involving students in action strategies. 



Environmental Education 
Goals and Objectives 

The ultimate goal of environmental educa- 
tion is to establish an en/ironmentally lit- 
erate and involved citi7.enr\'.- To meet this 
goal, it is important that students not only 
learn environmental concepts but also be 
able to define environmental issues within 
a social context, understand multiple per- 
spectives on an issue, and understand their 
roles in both environmental problems and 
solutions. (For a more in-depth discussion 
on the definition and goals of EE, refer to 
"Defining Envir(;nmental Education" in 
the Workshop Resource Manual.) 



Objectives 

of Environmental Education 

Awareness 
Knowledge 
Attitudes 
Skills 

Participation 

from the Tbilisi Declaration. Unesco ConlerencQ. J 5 77 ' 
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Environmental Awareness and Knowledge 
Two important factors that you can assess 
in an armchair review are accuracy and 
breadth of content. Teachers not familiar 
with an environmental topic area may 
need help in developing content specific 
criteria. You can start by narrowing the 
topic into 5-10 key concepts and develop 
criteria that address these concepts. You 
may wish to team up with environmental 
specialists for this step. Some important 
criteria might be: 

- terms are defined 

- environmental concepts are set 

in a social as well as an ecological 
context 

- ecosystems include individuals, 

the local community, and the global 
community 

- environmental issues relate 

to specific environmental concepts 

- information contains multiple 
perspectives lor bias is declared) 

- concepts are relevant to learners' life 

- depth of information is relevant 

for the intended developmental level. 

Attitudes 

There's no way around it, environmental 
issues and the solutions we choose are value- 
laden. Attitudes about the environment are 
based on individual and social value systems. 
It is important for an EE activity guide to 
help students define and understand their 
own values as well as understand and value 
other people's perspectives. Students ideally 
come away from a lesson feeling positive 
about participating in environmental issues. 
Considerations of attitudes include: 

- activities encourage a sense of personal 
stake and responsibilit>' 

- materials express a hopeful outlook 

- materials contain positive role models 
for discussion and action 

- activities encourage students to clarify 
their own values 

- environmental topics are covered from 

a variety of perspectives, and no one per- 
spective is predominant. 



Skills and Participation 
Students are more likely to become active 
in environmental solutions if they are 
aware of the range of possible actions, are 
able to evaluate them for appropriateness, 
and have the skills necessary to implement 
chosen action strategies. Successful EE 
instructional materials address action strat- 
egies in several ways: 

- environmental problems are presented 
with a range of possible solutions 

- the material compels students to 
consider the implications of possible 
solutions 

- activities help students gain nevv 
problem solving skills 

- action strategies focus on win-win 
solutions, whenever possible 

- activities provide students with 
critical reasoning and problem 
solving experience 

- both individual and community action 
strategies are presented 

- activities help students gain communica- 
tion skills, group skills, leadership skills, 
and other interpersonal skills necessary 
to become involved in solutions. 



Pedagogy 

Pedagogy, or how the material is taught, 
is an important consideration. Good envi- 
ronmental education is often said to be 
good education in that it emphasizes inter- 
disciplinary, experiential, relevant learning 
and focuses on preparing students to par- 
ticipate in their community. Because envi- 
ronmental education is as much a process 
as a content area, many pedagogical crite- 
ria overlap with the environmental educa- 
tion criteria. 

•- a variety of cultural perspectives 
is included 

- content excludes gender, race, age, 
and other biases (for more information 
on bias, see page 9) 

- different learning st>'les are included 
(for example, hands-on, student- 
centered, research, lecture, and other 
methods) 
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activities allow students to build from 
previous knowledge 

materials offer students the opportunity 
to practice critical thinking processes 
(such as hypothesis making, collecting 
and organizing data, analyzing, drawing 
conclusions, and problem solving) 

activities facilitate and encourage 
cooperative learning strategies 

teaching methods are appropriate to the 
guide's goals 

content pertains to state or local learn- 
ing objectives 

materials can be readily integrated into 
established curricula 

appropriate student assessment methods 
are included 

learning opportunities are conducted 
in a variety of settings 

lessons include suggestions for extend- 
ing or adapting the learning experience 

the materials can be easily adapted 
to varied learning environments, such 
as large and small classes, mixed-level 
classes, or rural and urban settings. 



A Bit About Bias 

Teachers should address the issue of bias 
when evaluating resources. Each of us has 
our own biases developed through our 
experiences. In addition to our personal 
biases, we are confronted with many forms 
of bias every day — in textbooks, posters, 
newspapers, and on television. Industry 
and environmental groups are most often 
accused of presenting slanted information, 



Freshkills, Staten island Is the 
world's largest landfill. It serves 
7.5 million people and wont be 
filled for another 10 years. 
Thirty-five similar landfills could 
serve the whole United States 
and occupy 0.005% of the land 
in the lower 48 states. It would 
take more than 7,00)0 centuries 
for the United States to be buried 
in its own solid waste. 



Presentation and Design 

If a guide has illegible type, few graphics, a 
binding that disintegrates, or an e,-;orbitant 
price, teachers may never discover that the 
content is good. Evaluating effectiveness of 
material design takes into account such ele- 
ments as text and illustration quality, and 
the guide's organization. 

- the goals and objectives of the materials 
are clearly stated 

- teacher instructions are clear 

- guide is easy to use and understand 

- handouts and other masters arc 
included and are easy to duplicate 

- the writing is clear and interesting 

- illustrations arc appropriate 
and engaging 

- required activiU' matcri \\s arc supplied 
or easily obtained. 



— NYC Department of Sanitation 

Although this is factual, one can detect 
bias. Where 35 enormous landfills 
will be placed, the cost of trucking 
waste from one state to another, and 
the wisdom of burying resources 
at current rates arent mentioned. 



but almost all forms of informational media 
contain some type of bias. Many are very 
subtle and merely a matter of how "facts" 
are determined and presented. Who pays 
for the material often indicates what slant 
■t might take. For example, textbooks tend 
to include the same material, edition after 
edition, because that is what teachers and 
curriculum committees expect and there- 
fore will buy. The bias is toward status 
quo and tradition, determined by sales. 
Other forms are more obvious, such as 
language that portrays leaders, scientists, 
or active individuals as men, or illustrations 
of domestic roles portrayed by women. 
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It is important, therefore, for educators 
to acknowledge that we do not live in a 
world free of bias and to facilitate stu- 
dents' ability to recognize and address bias 
in their lives. 

Recognizing bias is the first step to 
approaching issues (especially value-laden 
environmental issues) with an open per- 
spective. V, . ^ach have our own bias; com- 
paring different materials will help identify 
it. Activity 5: "What About Bias" (page 
43) is designed to help teachers see bias in 
instructional materials and discuss how to 
overcome it. 

Some types of bias relate to: 

- race 

- culture 

- age 

- gender 

- political views 

- economic status 

- different stances (e.g. industrialist 
vs. environmentalist). 



Identification of bias is a critical step in 
evaluating materials. You may not want to 
eliminate all biased material. Such material 
can help students understand what bias is, 
that they are affeaed by bias, and that they 
would, therefore, gain from identifying 
their own biases. Educators can deal with 
bias in a variety of ways: 

- not use the material 

- accept the bias as is 

- balance the bias with other materials 

- point out the bias to students and have 
them discuss and analyze it 

- help students develop critical thinking 
skills to identify bias on their own 

- modify materials to reduce bias. 
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Developing Your Evaluation Strategy 



ow DO YOU BEGIN to deal with all these criteria and how do you 
keep track of them when you are reviewing activity guides? In 
developing your strategy you will prioritize criteria, choose ques- 
tions that address these criteria, devise a way to apply the criteria 
that is meaningful, and keep track of your impressions so that you can compare 
several activity guides. 

Teachers will benefit most from a streamlined evaluation process and 
simple tools. This section, along with the evaluation tools included in Section 
VI, will allow you to develop an evaluation strategy that meets your own and 
teachers' needs. Section V provides several ideas for helping teachers person- 
alize this evaluation strategy in a workshop setting. 

For ease of understanding we have presented the evaluation process as 
four steps: Step 1, Quick Review of Materials (p. 11), Step 2, Prepare Your 
Evaluation Tool (p. 12), Step 3, Compile Results (p. 14), and Step 4, Evaluate 
the Evaluation (p. 15). You will probably go back and forth between steps to 
assure your strategy meets your goals. Keep in mind that you don't have to 
reinvent the wheel. Use this section to help you modify existing tools to best 
suit your needs. (Three samples are provided in Masters 3, 4, and 5.) 




Step 1 

Quick Review of Materials 

This first step is a fast way to get an , 
initial impression of the materials you 
will be evaluating. You will most likely 
notice prevailing presentation styles, 
glaring biases or omissions, and age- 
appropriateness concerns. A quick 
review is not necessary but is useful 
in the following ways: 

- To categorize nmterials into content 
areas, age levels, and other categories. 
One set of criteria may not be appropri- 
ate for all types of materials; you may 
wish to refine your criteria for each 
category. 

- To reduce the number of materials 
included in your evaluation. Once you 
have categorized materials, you may 
want to eliminate some categories from 
your evaluation. For example, you may 
decide to evaluate only those materials 
relevant to middle school social studies 
or to water quality issues. Be careful not 
to discard items because they are not 
familiar or do not match your particular 
style and viewpoint. 



- To review the material with an eye 
toward deciding how to apply criteria. 
For example, you may find that all 

of the materials fit basic EE goals but 
not in the same way. This would lead 
you to ask how well each of the goals 
are met. 

You can use the questions that follow to 
make your initial cuts. Questions of this 
type can most often be answered with a 
simple "yes" or "no" and still provide 
good direction about whether to take a 
closer look at the material. 

- Does the activity guide focus 
on a desired content area? 

- Are activities age-appropriate? 

- Are the activities appropriate for the 
learning setting (for example, outdoor 
center, classroom, factory visit)? 

- Is the guide readily available to teachers 
in your area? 
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Picking Strategy 

Sometime after they discussed developing an evaluation session for their 
next workshop, Anita and Steve gathered their teacher resource materials. 

"No wonder teacher* don't use all the resources we give them. No one 
has enough time to go through all these materials!" Steve sighed as he 
pushed aside a stack of activity manuals. "I know some of these are great 
resources only because I've used them. We need to have some system that 
will help teachers make choices without having to test each one." 

Anita glanced at Steve'o stack of resources. "OK, it's obvious we don't 
need some of these n'at'jrials for a biodiversity workshop. This one doesn't 
address biodiversity and here's another that only has ac' vities for early ele- 
mentary children. Let's weed out the ones that don't fit with the workshop." 

Anita and Steve began sorting through the materials selecting only those 
that pertained to biodiversity. After a few moments Steve noted that he had 
found quite a few resources addressing biodiversity but one was a comic 
book, two were videos, several were activity guides, and some were simply 
informational pamphlets. Using the same criteria to evaluate such varied 
media would be difficult. They decided to narrow their search to activity 
manuals that dealt with biodiversity. 



Step 2 

Prepare Your Evaluation Tool 

There are two ways to approach this 
step: use an existing evaluation tool or 
develop your own. While it may be more 
expedient to use an existing tool, you will 
want to modify it to meet your specific 
needs. Activity 3, (page 28) provides 
several sample tools. Use the ideas under 
approach A as a guide to modifying these 
tools for your use. Developing your own 
evaluation tool may seem like a daunting 
task. Approach B breaks this task down 
into substeps to make it more manageable. 
Remember, the shorter and simpler your 
tool, the easier it will be to use 

No matter which approach you take, 
remember that the easier the tool is to use, 
the more likely your results will be easy 
to understand. Here are some general rules 
of thumb when modifying or developing 
an evaluation tool: 

- Format so that the page is easy to read; 

guidelines are easy to follow 

- Make it easy to tally answers 



- Have questions relate to one criterion 

- Keep it short 

- Do not use double negatives 

- Do not ask two questions at once. 

A. Modifying Existing Tools 

Most evaluation tools were developed 
for evaluation projects that meet the needs 
of a large and varied audience. The criteria 
may be broad and the rating scheme com- 
plex. Before using them, consider adapting 
them to reflect the specific needs of the 
teachers in your area. If you are a bit 
uncertain about your teachers' concerns, 
you could meet with a few teachers c-r talk 
to a district curriculum coordinator. In a 
teacher workshop you may want to brain- 
storm what teachers need from EE materi- 
als (see Activity 1 ). To modify existing 
tools, you may cut, change, or add ques- 
tions, and modify how questions are rated. 
Here are some guidelines on modifying 
both criteria and rating schemes. 
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Both went back to work. In a short time Steve and Anita reduced their 
stack to a more manageable size. Antfa noted, "We could weed out a 
few more if we identified resources that deal with the appropriate age level 
Let's select only those that are designed for grades 4-6 or grades 7-9." 

"We could be even more specific and categorize them by learning setting 
like outdoor, classroom, or nature center," Steve suggested. 

This time Anita and Steve found their system was getting too complicated 
Few of the resources met all the criteria. They decided upon the following 
criteria to make their initial cuts: 

- activity manual 

- contain activities on biodiversity 

- designed for grades 4-6 or 7-9 

When they finished sorting and piling, Steve felt a sense of accomplish- 
ment. "I think we've saved ourselves a lot of time. Now that we've reduced 
the numbers, we can devote more time to evaluating them," he said. 

Anita pulled out a sheet of paper "You're right! Let's figure out what kind 
of things we'll look for as we evaluate." 



Narrow yoti/ scope: Balancing thorough- 
ness and simplicity is crucial to success. 
Start by eliminating criteria that do not 
relate ro your coHteni '-'.^riccrns. Then 
priorir!:7e the Ciiteria and focus on what 
vviii help teachers incorporate EE into 
their lessons. 

Adding Questions: You may find that 
you wish to add questions that apply to 
local education mandates or goals. When 
adding questions, frame them similarly 
ro existing questions so you may use the 
same ranking process. To make tallying 
responses easier, keep the ideal solution 
positive. Similarly, on a rating scale of one 
to five, always make five the best or the 
worst. 



Changing Criteria: Rather than eliminate 
a criterion, consider modif\'ing the way it 
is applied to make it more relevant. Tliere 
are several ways to ask a question about 
the same criteria. For example, you could 
change a general question about learning 
styles. "Are many types of learning situa- 
tions used?" to fit a specific learning style 
need; "How many lessons incorporate 
student-directed learning.'" You may wish 
to refine the questions to better relate to 
the materials you are evaluating or teachers 
needs. Use the same guidelines as above 
to write good questions. 

\lodif\'ing the rating system: Many tools 
are designed to give statistically reliable 
data. Therefore, the rating procedure 
may be more complex than you need. 
You may not even need a raring tystem 
unless you wish to compare among several 
activity guides. Instead you could make 
notes about the strengths and weaknesses. 



19 



ii(j.»f.-,' -fi-imrnf,!/ i,:ui,);-on Mitrn.il; 



B. Ci«ating Your Own Tool 

To develop your own evaluation tool, you 
must first create a survey instrument out 
of the criteria you huve chosen. Then you 
figure Jut how the answers to those ques- 
tions will help you compare activity guides. 
These two concepts go hand in hand. 

Convertiit^ Criteria: The criteria sets in 
Section II will help you thmk about the 
criteria that are important to you. To turn 
a generic criterion into an evaluation ques- 
tion, think about what you really want 
to know and how teachers will use the 
information. Questions can be asked to 
give ves/no answers, to help choose a num- 
ber on an answer scale, or to give a narra- 
tive response. For example, the criterion, 
"Activities encourage students to clarify 
their own values" may be converted into 
several questions: "Do activities guide stu- 
dents in identifying value positions held by 
themselves and others?" "To what extent 
do activities allow students to form an opin- 
ion?" or "To what extent are students 
encouraged to defend or challenge their 
opinions?" Framing the criteria will take 
some practice and you may wish to do so 
in conjunction with the next step. 

Developing a Rating System: The ques- 
tions must help you evaluate the materials. 
What will each answer mean? You can use 
anything from a simple checklist to a com- 
plicated ranking procedure. The following 
common methods may be appropriate for 
your use: 

- Cherk off which criteria are met. 
This works best with yes/no questions. 

- Each activity can be assigned a score 
and the scores averaged or added 
together to give each guide a total score. 

- If some questions are more important 
than others, think about weighting some 
questions in a summary score. 

- Use a narrative response style for the 
criteria. This will provide more informa- 
tion, but it is difficult when used to 
compare many activity guides. One way 
to present information is to group simi- 
lar responses together or to develop a 

short annotation for each activity guide. 

If more than one person is evaluating, 

you can indicate how often similar 

responses were given. 



- The most common type of tool provides 
a range of responses for each criteria. 
In this method, criteria measure hoiv 
well the guide meets each criteria, rather 
than a yes/no type of response. For 
example, "On a scale of 1-5, where 
1 = not at all and 5 = very well, how 
well does the activity guide incorporate 
different learning styles?" 

Most new evaluation tools will have a few 
kinks to be worked out. We recommend 
that you pilot test your tool on at least two 
activity guides. See if each criterion can be 
addressed the way it is asked. If not, try to 
phrase it differently. Also, make sure the in- 
formation generated by each criterion is 
useful in determining wh:;cher the activity- 
guide meets your goals and those of EE. 



Steps 

Compile Results 

If you are evaluating mav'rials simply for 
vour own use, you may not need to com- 
pile the results into a publication or report. 
You could simply keep the rating sheet 
with the activity guide or jot down notes 
on the strengths and weaknesses. If you 
do wish to compile the results for others 
there are several approaches: 

- List only the materials that have met 

a minimum number of criteria. Include 
a list of the criteria so that teachers 
can determine whether the materials 
have been rated on items that meet 
their needs. 

- Present the ratings in a matrix that 
show how each guide met the criteria. 
This allows teachers to select materials 
that rate high on criteria that are impor- 
tant for them. 

Write an annotation for each guide that 
shares the strengths and weaknesses. If 
you compile results for others, be sure 
to explain your evaluation methods 
and criteria. 
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Table 1: Sample Format for Compiling Results 



Water, Precious Water Overall Score = 4 

This collection of lesson plans is a well-organized series of laboratory 
hands-on activities. 

Report Card 

Teaching and Learning A- Grades K - 6 

Presentation and Organization A Disciplines:Math, Science 

Water Content B+ 

Teaching and Learning: The labs emphasize the scientific process 
of investigation and offer many opportunities for inquiry. Suggestions 
for extended activities are included. Most teachers will want to choose 
from the wide variety. 

Presentation and Organization: The directions to teachers are clearly 
written and contain well-defined objectives. The work pages are well- 
illustrated, uncluttered, and easily duplicated. The book's binding is a 
little weak. No bibliography. 

Water Content: The broad range of topics cover water awareness, 
the water cycle, water quality, water conservation, treatment and 
distribution, and the properties of water However, there is nothing 
on water resource problems in California. 

Teacher's Thoughts: Students and teachers both will enjoy 
working through these lessons which emphasize the quantitative 
nature of science. 

Adapted from Compendium for Water Resources, California Department of Education* 



Step 4 

Evaluate the Evaluation 

Think of your evaluation technique as an 
evolving proces.s. The more you reflect 
on the results, the more you will increase 
your skills and produce better results. To 
evaluate your evaluation, make sure that 
you have met the needs of teachers who are 
selecting HE activity guides. Indirect evalu- 
ation can not. however, guarantee that the 
materials will work with students. When- 
ever possible maintain records of teacher 
feedback on their experiences with the ma- 
terials in the classroom. Clomparing this in- 
formation to your evaluation results will 
help you refine your criteria and methods 
the next time you evaluate materials. 



Guidelines and Reminders: 

Aim to identify strengths and 
weaknesses of the materials. 

Establish a systematic procedure, 
follow, and document it. 

Include a Not Applicable option 
in your questions. 

Pilot test your tool. 

Check with teachers about their 
reactions and advice. 

Document your methods. 
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A Look at Formal Evaluation Projects 



IF YOU ARE USING someone else's evaluation results, you may wish to 
critique their findings. First, examine their work closely to make sure 
the evaluation is relevant to your situation. Find out how the data were 
gathered. If this information is not apparent or detailed enough, contact 
the authors. 

This section gives an overview of three common evaluation methods, 
described in order of increasing complexity. Each is illustrated with a real 
application to help you critically review the evaluation results, understand the 
strengths and weaknesses of each approach, and use the results of each method 
appropriately. As you review these evaluation strategies, be sure to note the 
methods used, built-in assumptions, and the format used to report results. 
These and the faaors listed below affect how applicable the results are to your 
specific needs: 

Evaluators: Do evaluators have relevant experience? Were they familiar 
with the activity guide's specific content and EE in general ? Do they have 
classroom experience? Do they represent the teachers who might use 
the evaluation results? 

Criteria: Were they chosen to meet specific audience needs or general 
audience needs? Are criteria meaningful to you? Are criteria that are 
important to you given a high priority? 

Results: Do the results provide enough information to allow you to 
make an informed decision? Do the recommendations take into 
account the context in which the guides will be used? 

Be war\' of many "evaluations" that include no critical analysis. They 
are usually reviews that list materials in categories, but without ratings. If you 
do an ERIC (Educational Resources Information Center) search for evalua- 
tions of EE instructional materials, you will find the majority are reviews and 
formative evaluation projects. Comparative, quantitative evaluations that rate 
EE activity guides are fairly rare. 
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The Expert Review 

The Expert Review method uses one or 
more select people to conduct an armchair 
review of instructional materials. The 
expert is most often an impartial external 
evaluator who has experience with the cri- 
teria. The expert must have both environ- 
mental education and content expertise. 

This approach offers several advantages. 
An external expert may provide an objec- 
tive look at materials that are produced 
internally. An expert review can provide c 
comparative evaluation of many resources 
in a short amount of time. This approach 
also offers a relatively low cost alternative 
to developing staff expertise in EE and 
evaluation. When using the recommenda- 
tions from this type of review look for: 

Bias - The biggest disadvantage to the 
expert review model is the limited per- 
spective one or a few persons can bring 
to the task. You run the risk of a heavily 
biased evaluation. 

Qualifications of Experts - It may be 
difficult to find one person with both con- 
tent and educational expertise. You may 
wish to research the qualifications of the 
experts if they are not listed. 

Support for Recominendations - This 
method yields an educator's interpretation 
of what will be instructive for students. 
Actually using the materials may yield dif- 
ferent results. You may wish to find first- 
hand knowledge of some of the materials 
to double check rhe recommendations of 
the expert. 

The Pacific Gas and Electric 
Teacher's Resource Guide, 
Sacramento, California 

Pacific Gas and Electric (PG&lE) recog- 
nized that the utility needed to evaluate 
the materials they disseminate to educa- 
tors before ordering new materials and 
printing another catalog. Since they did 
not have an education specialist on staff, 
the company opted to condr.t an evalua- 
tion by contracting with an EE specialist 
familiar with energy issues. The expert 
they contacted recendy compiled a com- 
puterized collection of more than 300 



energy education resources and had writ- 
ten an energy education activity packet 
for grades 4-6.'' The expert spent three 
days evaluating the utility's library of 
materials based on criteria used in other 
evaluation projects. As a result of this 
effort, the expen recommended which 
materials should be rewritten, expunged, 
reordered, or added to the collection. 
Based on these recommendations PG&E 
rewrote its teachers' guide. 



The Group Review 

The Group Review is an expansion of the 
expert review. It uses a group of content 
experts, educators, and/or environmental 
educators. Their results can be tallied and 
statistically analyzed to say with qualified 
certainty that some materials are better 
suited to the chosen goals than others. 
Ideally, the evaluators repiesent a specific 
audience, provide expertise across the 
diversity of materials, and provide a geo- 
graphical, cultural, and gender mix. This 
approach is also an indirect evaluation 
although some of the evaluators may have 
first-hand classroom experience with the 
materials. Things to look for when review- 
ing this type of evaluation project include: 

Evaluators - Do they represent the spec- 
trum of teachers that come to your work- 
shops.' Does the mix of evaluators give 
the breadth of expertise that represents the 
range of materials evaluated.' 

Number of Evaluators - If there are not 
enough evaluators and responses don't 
agree, it will not be possible to interpret 
data with reliability. Resulting data could 
be misleading. The number of evaluators 
should be noted along with the degree of 
agreement between evaluators (often dis- 
played as a standard deviation). 

Methods - It is important that the evalua- 
tion tool was tested to assure that all eval- 
uators were evaluating in a similar fashion, 
interpreting criteria rhe same, and that all 
evaluations were accurately recorded and 
compiled. 

Results - Are the criteria reported with the 
results? How were the criteria determined.' 
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The United States Department of Agricul- 
ture Extension Service gathered a review- 
team of water education specialists to eval- 
uate water education materials. Sixteen 
people from state and private agencies 
combined their efforts to (among other 
goals) summarize information about water 
education materials. The project focused 
on nonformal education materials. The 
resulting guide was developed to help 
development of Cooperative Extension 
education programs and materials. 

This Water Quality Initiative Team sup- 
plied copies of instructional materials and 
recommended other resources. They also 
helped identify water education goals and 
topics. It was determined that due to the 
volume of water education materials u 
would be expedient to develop a short-term, 
initial project. The objective was to review 
and classify a selection of available materi- 
als as a basis for understanding what was 
missing and needed. 

An initial selection for materials 
to review was based on three criteria. 
Material was included in the review if it: 

- addressed one or more of the study's 
general or snecific goals 

- presented a planned education 
experience 

- improved representation of regional 
water concerns, varied water topics 
or environmental education goals. 



The chosen materials were evaluated 
against eight water topic criteria, five 
EE goals criteria, and five format criteria. 
The intent was to see whether the water 
topics, EE goals, and preferred format were 
present. The results were compiled to show 
overall strengths and weaknesses in each 
instructional material. A summary of each 
material provided information that will 
help educators select materials. The final 
report presents information in four ways: 

- an alphabetized listing of materials 
that provides a brief description of the 
material. 



- a listing of avail 
or region. 



abilitv bv state 



- a listing of materials from national 
organizations or nationally available 
sources, and 

- a matrix that shows review results. 



The Pre-Test/Post-Test Review 

Instructional materials sometimes fall 
short of expectations. Directly evaluating 
the impacts of the activities is the most 
accurate way to determine whether the 
materials meet the learning objectives. 
The pre/post approach is the only one 
described here that evaluates materials 
through direct interaction with students. 
This model dictates that students be 
given a test before they have a planned 
learning experience with the activities to 
be evaluated. After the lessons, students 
take a post-test, which is similar to the 
first test. Presumably an improved score 
means that the learning experience was 
a success. A control group completes the 
pre-test and post-test at the same time 
as the treatment group to assure that the 
instructional activity, and not another 
learning situation, was the source of the 
improved scores. Any increase in the con- 
trol group s post-test score results from 
a factor other than the instruction; this 
increase should be considered when com- 
puting the treatment groups gain. 

Teachers chosen to evaluate the activi- 
ties are randomly assigned to treatment 
and control groups. Activities are random- 
ly assigned to help increase accuracy of the 
evaluation results. The pre/post test meth- 
od is not usually used to compare among 
.ictivity guides: it is most often used to 
measure the effectiveness of a single set 
of activities. 

When reviewing the results of this 
type of evaluation, be sure to consider the 
following: 

Test Construaion - In using a pre-test, 
vou run the risk of alerting the students 
to the important concepts or simply raising 
their curiosity about the new material. 
The tests should be similar but not exact 
or students may improve due to praaice. 

Biases in Teaching - Teachers who are 
familiar with the activities could do a bet- 
ter job at teaching than ones that are not. 
Activities should be randomly assigned 
so teachers get a mix of familiar and new 
activities. 
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Significance of Learning - Success is typi- 
cally defined as a better post-test score, 
based on the activities that are conducted. 
This will not guarantee that all activities 
are equally successful, or that broad goals 
for the materials are met that are not 
addressed in these activities. 



Project Leamittg Tree, 
l.nvironmental Education Pre K-8 
ActitntK' Guide' 

Pre/Post testing was the primary method 
used to do a summative evaluation of the 
1994 Project Learning Tree (PLT) Activity- 
Guide. PLT contracted with the North 
American Association for Environmental 
Education (NAAEE) to evaluate the effec- 
tiveness of the new activities. The project 
entailed several professionals and a budget 
of over $50,000. 

Pre-tests were given in 120 classrooms 
in the grade-level groups of Pre K-1, 2-3, 
4-6, and 7-8. (Pre K -1 students performed 
tasks and their teachers reported results 
on computerized answer sheets.) The tests 
were designed to measure students' knowl- 
edge and attitudes about one of several 
themes, including diversity, interrelation- 
ships, patterns, structure and scale, and 
systems. About two days after administer- 
ing the pre-test, teachers in 60 classrooms 
used between three and seven randomly 
assigned aaivities that addressed one of 
the five strands. Within a week or two, 
students in all 120 classrooms took the 
same test again. All of the test data from 
a single classroom yielded a single score 
for that class. All the classes from the same 
grade level that experienced activities ad- 
dressing the same PLT theme were aver- 
aged into one score. Teachers were also 
asked to keep a log, noting anything that 
seemed significant as the evaluation pro- 
gressed. Logs are not always included in 
this evaluation model, but can provide 
some significant and helpful information. 
This qualitative information helps the re- 
searchers interpret results more accfately 
and can provide new insights into the use- 
fulness of the materials. 
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What to Do in a Workshop 

NE OFTHE BEST WAYS to build teachers' skill is to have them develop 
a survey tool and use it to evaluate some materials. If you do not 
have the luxury of a few extra hours, even a short overview on 
evaluation can be a valuable addition to any EE workshop. Below 
are three scenarios to help \'ou plan your workshops. The activities are 
provided in Section VI. 




Approach A 

Provide Evaluated Materials 
and Criteria 

Providing evaluated HE resources will 
reduce the effort teachers spend on select- 
ing instructional maccrials. Although this 
method often requires more of your time 
and work before the workshop, it is one 
of the simplest workshop strategies. If 
you familiarize participants with the goals, 
criteria, and results of evaluations, they 
can use this information to review other 
instructional materials. 

You can either use resources that have 
been evaluated or evaluate the ones you 
have, if you present materials that have 
been evaluated make sure the criteria, 
methods, and results are relevant to your 
teachers. Be sure to provide and discuss 
these elements. If you evaluate the materi- 
als, consider using one of the evaluation 
tools supplied in Activity 3, page 28. 
(Sections H and III can help you modify 
these tools.) When using this method to 
plan a workshop, consider the following 
tips: 

- Before the workshop, identify some of 
the teachers' key needs for resources and 
the areas of HE they want to address. 
Use these needs to modify the evaluation 
tools. 

- Reserve time in the workshop for teach- 
ers to describe and discuss the resources 
that have worked for them. 

- Consider providing workshop partici- 
pants with the actual evaluation tool 
that was used to evaluate the workshop 
materials. With the tool available, they 

I an decide which criteria nri.' most im- 
portant to them. 



Example Agenda 

I. Distribute handout listing evaluated 
EE instructional materials 

II. Overview of evaluation method 

A. Criteria used to evaluate 

B. Results of evaluation 

III. Teachers share favorite materials 



Approach B 
Developing Criteria 

The workshop strategy described in 
approach A puts you in the position of 
doing most of the work. Although this 
strategy saves time for teachers, empower- 
ing them with a hands-on evaluation expe- 
rience may increase their commitment to 
evaluating other EE instructional materi- 
als. This approach builds teachers' interest 
in evaluation and helps them understand 
how to best choose criteria that meet their 
needs. You might consider the following 
suggestions: 

- Build a rationale for evaluating HE 
materials. Using Activity 1 on page 24, 
help teachers think about vhy they 
would bother evaluating activity guides. 
Pair this activity with one of the two 
ideas below to give teachers the e.xperi- 
ence of developing their own evaluation 
criteria. 

- introduce the topic with a short over 
view of evaluation and the goals of 
EH. You can provide them with a 
short vc' sion of the information on 
pages 4 After this discussion, 
have teachers brainstorm and prioritize 
a list of criteria. You may augment 
this list with Master 2. 
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Alternately, get teachers thinking 
about evaluating criteria by having 
them review an activity guide and ask 
them tor their opinions. Help them see 
how their "gut reaction" can help gen- 
erate criteria. Have teachers work in 
groups to develop criteria for EE 
goals, pedagogy, and presentation. You 
may refer them to Section II. Activity 2 
details this strategy. 



Example Agenda 

I. Build a rationale (Activity 1) 

II. Overview of evaluation concepts 

A. Levels of evaluation 

B. Indirect vs. Direct 

III. Develop criteria (Activity 2) 

IV. Teacfiers sfiare tfieir results 

V. Teacfiers sfiare tfieir favorite materials: 
explain why they work 



Approach C 

Do a Mini Evaluation 

One of the best ways to prepare teachers 
to evaluate potential resources is to have 
them evaluate an activity or a guide during 
a workshop. While this approach is time- 
consuming, it provides teachers with the 
skills and practice necessary to become 
proficient evaluators. Given teachers' 
schedules, this may be the only time they 
actually get to evaluate some of the mate- 
rials you give them. 

Building on the activities suggested in 
the approach above, take teachers to the 
next step. Activities 3 and 4 provide strat- 
egies for leading teachers through an eval- 
uation process. As you de/elop the 
workshop, consider these ideas: 



Give participants an opportunity to 
develop their own criteria. You can 
streamline these strategies by having 
teachers review the EE guides while 
asking: How will students react to 
the activities? What will the activities 
help them learn? How will the activit>' 
guides affect student behaviors and 
participation in environmental issues? 

Divide into groups and have each 
group evaluate the same activity guide 
but with different tools. Discuss the 
strengths and weaknesses of the tools. 





" One of the most predous 
gifts you can give to teachers 
is time to plan how to use the 
new information tf ,«y acquire ■ 
from an inservice woricshop." 

Margaret Pennock 

Teacher, Mast Landing School. Freeport, 
Maine 

You can use this feedback to mod- 
ify the tools. 

Divide teachers into groups. Each 
group can evaluate a different guide 
with the best tool. Compile the results 
from group reports. 

Provide time for teachers to talk about 
which resources have worked well for 
them. 



m 
f 



~3 



■0. 



V 1 



•s. 



ERIC 



/ ,,j/U(i(inr f "I 'odmi'Tfcii to'i, J[ l O W.itw/ii/;, 



27 



mi 



Example Agenda 

I. Build a rationale for evaluation (Activity 1) 

II. Brainstorm the goals of EE instructional 
materials 

III. Develop criteria (Activity 2 ) 

IV. Evaluate an activity or activity guide 
(Activi'.y 3) 

A. Groups do same activity guide 

B. Groups 00 different activity guides 

V. Compile results 

VI. Discuss how to adapt tools 

VII. Teachers share their favorite materials 



m 
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Summary 

There is nn shortage of EE materials from 
which to choose. Only yoa can really 
determine which instructional materials 
best meet your needs and the needs of the 
teachers you serve. Helping you make 
those decisions accurately and quickly is 
a major goal of this unit. We have looked 
at some of the general principles of evalua- 
tion so you might create evaluation strate- 
gies that yield accurate and useful 
information. In this process we have dis- 
cussed selecting criteria and how to best 
apply them to the materials you wish to 
evaluate. Interpreting your results and 
evaluating your process are the final steps 
of this procedure. We also hope you gained 
some skills to interpret the result of the 
more formal evaluation efforts you will en- 
counter. 

If you find this systematic evaluation 
process valuable, you will want to share 
it with the teachers with whom you work. 
The background information, the work- 
shop guidelines, and the activities provided 
will help you encourage teachers to criti- 
cally evaluate the instructional materials 
they have and will encounter in the future. 
Ideally, we will all become informed con- 
sumers and disseminators of high-quality 
HK instructional materials. 
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Activities and Masters 




HESEWORKSHOP ACTIVITIES AND SAMPLETOOLS will help you design 
a short segment or develop an entire workshop that focuses on eval- 
uating EE materials. Using Section V as a guide, you can build these 
aaivities into a module that fits your participants' needs. 



Activities 

The first two activities help develop an overview of the topic of evaluation by 
defining the goals of evaluating materials and what it means to evaluate. 



Building a Rationale 

This activity helps participants justify 
an armchair review of EE materials. 
Teachers discuss benefits and goals 
for evaluating instructional materials. 



What is Evaluation? 

To introduce teachers to the concept 
of evaluation, this activity asks 
teachers to develop a framework for 
evaluating EE activity guides. The 
overview of EE goals and evaluation 
criteria categories will help teachers 
use evaluation tools and results 
of evaluation efforts effectively. Two 
masters for overheads are included. 



Immersing teachers in an evaluation project will help them gain both a better 
understanding of the process of evaluation and confidence in their ability to 
effectively evaluate new materials. Activities 3, 4, and 5 will also leave teachers 
with a set of strategies and tools that suit their particular needs. 



Trying Out Some Tools 

Using three sample questionnaires, 
participants critique and modify an 
instrument to help identify materials 
that meet their needs. 

Learning to Look 

Teachers will compare activities that 
contain similar content but different 
teaching methods. Participants will 
_^_distuss aspects of pedagogy that will 
affect evaluations of these types of 
instructional materials. 



0What About Bias? 

Teachers discuss the inevitability 
of bias, then practice discovering 
it in materials. Five strategies for 
dealing with bias are provided. 
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Activity ^ 



Building a Rationale 



This activity helps 
participants justify 
an arnnchair review 
of EE materials. 
Teachers discuss 
benefits and their 
goals for evaluating 
instructional materials. 



Objectives 

To help teachers identify goals for 
their evaluation. 



Materials 

Markers, tape. 

Time 

30 minutes 



m 



Outline 

Set the stage by asking teachers to imagine 
that they will be given a complete set of 
new resources for the school year. Although 
they are not familiar with these EE materi- 
als, they are all good teachers and know 
what works in the classroom. They will be 
using their expenise to evaluate these new 
materials, and to determine how to use 
them. But first, they should discuss what 
they will be looking for. 



Divide panicipants into smaller groups 
of three to five. Ask them to appoint a 
facilitator, recorder, and timekeeper. Give 
each group a piece of newsprint and a 
few markers. 



Give the groups 15 minutes to come up 
with a list of goals for EE instructional 
materials and record them on paper. You 
might find these questions help stimulate 
good discussion: 

- What do you want teaching materials 
to accomplish? 

- >XTiat do you look for in a piece 
of instructional material? 

- What makes activities a success 
with students? 

- Wh.ii do you need from new materials 
to help you teach well? 



Have each group post their report on the 
wall. Invite all participants to walk around 
the room viewing each others' goals. 



Conclude by leading a large group discus- 
sion on the similarities and differences 
of each group's goals. Invite panicipants 
to think about the value of taking time 
to evaluate materials by asking them the 
following questions: 

- What arc the basic goals of evaluation? 

- How can evaluation help you meet your 
teaching goals? 

- How can you use this exercise to help 
you think about evaluation? 



30 



Activity O What is Evaluation? 



To inlroauce teachers 
;o the ccnceot of 
■A'aluation, this 
.ictivitv asKs teachers 
;o develop a frame- 
.'.orK fcr evaluating 
EE activity guiaes. 
An overview of EE 
'oa/s ana evaluation 
, nteria categories 
:,iil heip teachers 
.ise evaluation toois 
;na the results of 
evaluation efforts 
■nore ef^ectivelv. 



Objectives 

To introduce teachers to the concept 
of evaluation; to provide a framework 
for evaluating EE activity guides. 



Outline 

Divide participants into groups'of three 
or four You may group teachers randomly, 
or by grade level or content specialty. 
Explain to the teachers that t^-^y will eval- 
uate activity guides. 



Give each group copies of an activit}' 
guide. NOTE: If you wish to have teach- 
ers bring their favorite activit}' guides to 
evaluate, ask them to work in pairs, using 
one guide between them. 



(jive the groups 10 minutes to quickly 
review the guide. Have teachers indepen- 
dendy list their impressions of the activity 
guide, both good and bad. Teachers then 
share their impressions with their group 
or partner Each group should record 
their reactions to the material. 



Have each group report their results. 
Record their comments on a newsprint 
or overhead. Lead a discussion using 
the following questions: 

- How were their results different? Why.^ 

- t)n wiiat grounds do they base their 
reaction to the material.^ 

- If they had more time, what else would 
thev consider? 



Materiak 

A set of activity guides for each group 
of three to four participants, news- 
print, markers, and overhead Masters 
1 and 2, if you wish. 

Time 

30 minutes 



Using overhead Master 1 and the back- 
ground on pages 4 - 5, explain the levels 
of evaluation. Identif\' which of 
their reactions relate to each level. 



Using background from pages 7 - 9 and 
overhead Master 2, provide an overview 
the three categories of criteria: EE goals. 
Pedagogy, and Presentation and Design. 
Discuss what criteria the groups used. You 
can stimulate the discussion by asking: 

- What questions did you ask yourself? 

- On which aspects of the material did 
vou focus? 



Have each group put their evaluation 
criteria into the categories. Have them 
discuss any gaps they see and develop 
additional criteria to fill these gaps. 
Have them review the activity guide a 
second time to rate how it meets their 
new criteria set. 

6 

(Conclude by asking teachers to discuss: 

- How is evaluation different from 
judging on opinion alone? 

- How can they apply this knowledge 
to looking at new HE materials in the 
future? 
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Master Q Lcvels of Evaluation 



Reaction Evaluation 

measures immediate response 

to the learning experience (students) 
to the materials (teachers) 

Learning Evaluation 

looks for changes in: 
knowledge 

awareness of environmental issues 
attitudes 

Behavior Evaluation 

looks for changes in: 

students' skills 
student behaviors 

Results Evaluation 

measures impact on: 

the environment 
students' lives 
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Master Q Criteria Categories 



Goals of EE 

• Awareness 

• Knowledge 

• Attitudes 

• Skills 

• Participation 

Pedagogy 

• Teaching and Learning 

- interdisciplinary 

■ age appropriate 

- critical thinking 

■ cooperative learning 
• assessment 

• Bias 

• Integration into Curriculum 

Presentation and Design 

• Format 

• Writing 

• Appearance 

• Organization 
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Activity O Trying Out Some Tools 



. :, r..7 Samoie 
:L.est:Gnnaires. 
.j'i.c.Cj/its cr.tiaue 
'ems ana moaifv 
: " :-"jumeni to 
■■-■■- r, ysrn idcntifv 
• iier'a/s that meet 



Ob;cctrt;es 

To help participants recognize the 
importance of how evaluation ques- 
tions are worded and the length of 
evaluation tools. To give participants 
practice in using several evaluation 
tools. To have participants modify 
an instrument to meet their evaluation 
needs. 



Materials 

One activity guide for each group or 
copies of activities. Copies of each of 
the sample evaluation tools provided 
in Masters 3, 4, and 5. 

Time 

One to two hours with optional 
activities 



- . / - ! 
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Outline 

Participants will practice using different 
evaluation tools to evaluate the same piece 
of instructional material. This will give 
them insight into how different criteria, 
questions, and rating strategies can affect 
the results. They will use this insight to 
adapt the tools for their use. 



Distribute a copy of each sample tool and 
either one activity or a complete activity 
guide. Ask each team to evaluate their 
material with each of the three tools. Have 
them discuss among themselves which tool 
they liked best and why. 



When teams finish their reviews, lead a 
discussion with the following questions: 

- Did the evaluations result in equivalent 
information? 

- If nor, how did they vary? 

- On which criteria arc the answers 
most varied? Why? 

- What questions did they have about 
using the tools? 

- Which tool did thev like best and why? 



3 

(optional) 

Have each group list the characteristics 
of the tools they liked best. Ask them to 
consider the types of questions, rating 
strategy, ease of use and how it related to 
their teaching needs. Using these qualities, 
have the group modify one of the tools 
or develop a new one based on their needs. 

4 

(optional) 

Have each group share their new tool, 
presenting the characteristics, types 
of questions, and comparison strategy 
to the whole group. If possible, make 
copies of each new tool for all groups. 
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Master 

B The EE Curriculum Inventory Tool 



Directions: Indicate how well the inventoried materials will help 
students to know about or accomplish the statements for each goal level. 
Circle the appropriate number for each statement according to the scale 
below. Compute the Goal Level Average as directed below. 



Evaluation Scale: Notatall Poorly Slightly FairlyWell VeryWell 
0 12 3 4 



Goal Level 1: Ecological Foundations (Knowledge) Rating 

1. Populations: The nature/behavior of natural populations 0 12 3 4 
(birth, growth, change & its effects, death, extinction, etc.) 

2. Natural Communities: Structure behavior, interaction 0 12 3 4 
of Individuals and populations in a particular place (forest, 

desert, sea, pond, etc.. food chain, food web, predation, etc.) 

3. Ecosystems: Structure, interaction, and effects between 0 12 3 4 
living things and the non-living enviroiiment (natural change, 

succession, adaptation, etc.) 

4. Energy & Chemical Cycles: How energy chemicals, and 0 12 3 4 
pollutants go through a natural system (water, oxygen, 

energy carbon, etc.) 

5. Balance of Nature: The tendency of natural systems to estab- 0 12 3 4 
lish balance over time when disturbed naturally or by human 

pollution. 

6. Humans as a Component of Natural Systems: The effect 0 12 3 4 
of human behavior on natural systems. (Development of 

technology effects of people's attitudes, and values.) 

Ecological Foundations Total: Add all circled numbers. 

Ecological Foundations Average: Divide by 6. 
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Goal Level 2: Issues and Values (Knowledge) 

7. Identifies the difference or discrepancy between an environ- 
mental problem and issue. (Environmental problems may 
be natural or human-caused while issues are the conflicts 
arising from how people view the problem). 

8. Communicate the way people's culture affects the environ- 
ment (politics, society, religion, etc.). 

9. Communicate the way individual people affect the environ- 
ment (individual actions and behavior). 

10. Identifies (clarifies) one's own attitudes and values toward 
an issue or the environment. 

11. Communicates that there may be more than one way 
to resolve an environmental issue. 

12. Communicates that the difficulty in resolving environmental 
issues stems from the different attitudes and values of the 
people involved in them. 

Issues and Values Total Add all circled numbers. 

Issues and Values Average Divide by 6. 



Goal Level 3: Investigation and Evaluation (Skills, participation) 

13. Uses problem-solving skills to identify specific environmental 
problems and related issues. 

14. Identifies the people (actors) involved in an environmental 
issue. 

15. Identifies the beliefs and values that explain people's 
positions on an environmental issue. 

16. Examines a variety of environmental issues and their 
solutions. 

17. Identifies the ecological costs and benefits of designated 
solutions to environmental problems and issues. 

18. Identifies the human costs and benefits of designated 
solutions to environmental prob ems and issues. 

Investigation and Evaluation Total: Add all circled r 

Investigation and Evaluation Average: Divide by 6. 



Rating 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 



Rating 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 
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Goal Level 4: Environmental Action Rating 
(Knowledge, skills, participation) 

19. Communicates the need for responsible citizen action 0 12 3 4 
to resolve environmental issues. 

20. Communicates that there are various levels of environmental 0 12 3 4 
action (individuals, groups, organizations). 

21. Identifies examples of environmental action categories 0 12 3 4 
(persuasion, consumerism, political action, legal action, 
ecomanagement). 

22. Examines scenarios/case studies that would allow the 0 12 3 4 
student to apply knowledge of environmental action and 

in choosing responsible environmental action. 

23. Identifies the human and ecological costs and benefits 0 12 3 4 
to identified environmental actions. 

24. Takes individual and/or group action on an environmental 0 12 3 4 
problem they have identified and analyzed. 

Environmental Action Total: Add all numbers circled 

Environmental Action Average: Divide by 6. 

Goal Level 5: Teaching Methods and Practice Rating 

25. Do the materials provide specific training in the use 0 12 3 4 
of problem solving skills such as observing, inferring, 

predicting, classifying, hypothesizing? 

26. Do the materials provide opportunities to apply problem 0 12 3 4 
solving skills with simulated models, real life situations, 

scenarios, etc.? 

27. Are the materials educationally appropriate for the intended 0 12 3 4 
student users (reading level, conceptual and skill levels, etc.)? 

28. Would the materials be attractive and appealing enough 0 12 3 4 
to gain students' interest? 

29. Do the materials contain goals and measurable objectives? 0 12 3 4 

30. Do the materials provide sufficient suggestions or mecha- 0 12 3 4 
nisms for evaluation? 

31. Is the information provided for students and teachers 0 12 3 4 
accurate? 

32. Do the materials provide sufficient background information 0 12 3 4 
for the teacher? 

33. Do the materials provide clear directions and suggestions 0 12 3 4 
for teaching? 

Teaching Methods Total: Add all numbers circled 

Teaching Methods Average: Divide by 9. 
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Summary Data 



Evalu^tor's Name 



Date 



Title of Curriculum Materials 



Author 



Publisher 



Copyright Date 



Summary Graph 

Directions: You may present the results of your evaluation on the 
graph below (with the averages on pages 1. 2. and 3.) Several 
people may also choose to combine and average their scores for 
each curriculum so that a group score can be graphed. Shade the 
bar that corresponds to each goal. 



4 — 



Ecological Issues 
Foundations and Values 
Average Average 



Investigation 

Evaluation 

Average 



Environmental Teaching 
Action Methods 
Average Average 



Adapted and used With oetmii'.iijr). J.R. Gordella. 1993 HiRrtMon Hpiprtts. KY: 
Scrtoo/ ol tducMion. Northern Kentucky tjmversit', 
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Master 



Development and 
EE Resources Tool 



Resource: 



Rate As: ' 1" Meets criterion 

"2" Some attention to criterion 
"3" No attention to rriterion 
"NA" Not Applicable 

Knowledge Criteria 

Resources should contain essential elements of core concepts. 

Development Issues 

Sustainable Development 

Presents the need for balance between economic growth and 
social development in the interests of sustainable develop- 
ment and overcoming poverty and social injustice. 

Environment and Development 

Presents the importance of ecological balance against the 
effects of environmental destruction and the need to balance 
economic growth and conservation through a process of 
sustainable development. Presents the importance of biodi- 
versity T-.d sustainability. 

Rights and Development 

Presents the individual rights and responsibilities as citizens 
of local, national, and global communities. Supports the need 
to eliminate discrinination and other threats to human rights 
from various sources. Presents the rights of all species as 
essential for biodiversity and sustainability 

Green Aid 

Presents aid projects that promote economic development in 
an environmentally responsible manner. Presents benefits to 
both donors and recipients. 

Global Perspective 

Presents the interdependence of global economic, social and po- 
litical systems. A selected global perspective reflects the princi- 
ples of economic justice, human rights, sustainable development 
and peace. 



Rating 

1 2 3 NA 

1 2 3 NA 
1 2 3 NA 



1 2 3 NA 



1 2 3 NA 



1 2 3 NA 



Comments: 
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Critical Skills Criteria 

Resources should help participants challenge bias, support ratio- 
nal decision making, examine solutions and prospects for change. 

Balance Issues 



Rating 



1 2 3 NA 



Comprehensiveness 

Includes a wide range of social-cultural viewpoints. Sensitive 1 2 3 NA 
to explanations of social structure. Can distinguish between 
description and exianation. 

Stereotyping 

Presents a range of roles and human characteristics for any 
group. In-depth portrayals of sub-groups include accurate 
presentations of the group culture, from the point of view of 
that group. Sub-groups portrayed active participants in their 
community in a variety of decision making situations. 

Misrepresentation and Validity 

Does not present unsubstantiated 'factual' claims as 'truths.' 1 2 3 NA 
Sources are verifiable, accurate, and up-to-date. Does not 
include misleading implications. 

Technology 

Does not imply that the world's major problems can be 1 2 3 NA 

solved by further development of industry and technology. 
Identifies technology as part of the technical-social complex 
of productive forces. Technological solutions to social issues 
are seen in their political, economic, and social context. 

Interdependence 

Presents an issue in its connection with the wider ecological, 1 2 3 NA 

economic, political, and social environment. Presents individ- 
uals and communities as linked. 

Decision Making 

Presents conditions for rational decision-making method. 1 2 3 NA 

Emphasizes inquiry process of hypothesis and generalization, 
cooperative learning, critical reflection, dissemination,and 
involvement and action. 

Problem Solving 

Evaluates the relevance, validity, and implications of alterna- 1 2 3 NA 

tive solutions. Presents the interests and values underlying 
various solutions. Supports a multi-disciplinary approach. 

Comments: 
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Attitudes Criteria Rating 

Resources should help promote understanding and solidarity 1 2 3 NA 

among all peoples. 

Empathy 

Sensitive to the viewpoints and feelings of other people. 1 2 3 NA 

particularly those belonging to groups, cultures, and nations 
different from our own. 

Respect 

Supports the development of self-respect and respect 1 2 3 NA 

for others, particularly those with national, cultural, and 
family background different from our own. 

Participation 

Supports solidarity with victims of injustice in one's own and 1 2 3 NA 
other societies. Supports participation in political decision- 
making at local, national, and international levels. Supports 
solutions of ma)or world issues through cooperation at levels 
of individuals, organizations, and nations. 

Comments: 



Values Criteria Rating 

Resources should help participants develop a commitment to 1 2 3 NA 

working with and for others in the interests of equality, equity, 
and sustainability. 

Social Justice 

Examines inequalities of wealth and power Presents efforts 1 2 3 NA 
to redress them through development, aid, trade, and equity 

Equality of Peoples 

Presents the principles of equality and den-iocracy which 1 2 3 NA 

underlie a belief in the equality of all peoples, as the right 
of all peoples to life, freedom, and self-determination. 

Ecological Sustainability 

Presents biodiversity as a right of all living species and as a 1 2 3 NA 
basic principle of sustainability Presents intra-generational 
and inter-generational equity as a right of present and future 
generations and as a basic principle of sustainability 
Presents the satisfaction of essential human needs and aspir- 
ations as a priority and as a basic tenet of development. 

Comments: 



Used W(th Dermiiiion jna adapted from 'An<)//^mg Education Resources' bv Zita Uni^er. J 992 tn Teac/iinp lor a 
!^u!.tAinatitF! World: FMviror^mental anr i ppi/p/nfimpnf fdacntmn Pinip.c.t Inr Tpacher iducation. Johr) fien. editor. 
Brfibane. Australia: Gnllith University Press. 
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The following evaluation tool used by the teachers to rate nnaterials 
was adapted fronn the California Science Framework. Chapter 8, 
■'Instructional Materials Criteria." 



a b c d e^not applicable 

excellent poor 



A. General Content 



□ 


1. 


Are ideas expressed through unifying thennes and big ideas, without facts? 


□ 


2. 


Is content interdisciplinary? 


□ 


3. 


Are students challenged to utilize higher-level thinking processes 
(i.e.. inferring, relating, and applying)? 


□ 


4. 


Are ideas presented logically and connected through the curriculunn? 


□ 


5. 


Is depth of understanding ennphasized (rather than encyclopedicbreadth)? 


□ 


6. 


Are historical, ethical, cultural, geographic, econonnic, and 
sociopolitical relationships addressed? 


□ 


7, 


Are knowledge and learning shown to be connected to students' lives and society? 



B. Presentation 

I I 1. Are instructional nnaterials clearly and engagingly written with the nnain concepts 
well articulated? 

i I 2. Are the roles of environnnental ethics, citizenship, and stewardship explored? 

I I 3. • Do lessons pronnote respect and caring for the environment, 

yet be non dognnatic and open to inquiry and differences of opinion? 

I I 4. Are personal and societal values and conflicting points of view explored in context? 

I I 5. Are instructional nnaterials easy for students to use and understand? 

I I 6. Is learning nnade accessible to all students? 

I I 7. Are writing and concepts developnnentally appropriate for the 

designated grade, yet sensitive to individual differences in educational 
experience and learning mode? 

I I 8. Is environmental responsibility modeled in design, underlying 

philosophy, and suggested activities by the lessons and materials 
(e.g. using recycled materials and properly disposing of wastes)? 

I I 9. Are there clear linkages presented between communities of all levels 
("thinking globally, acting locally")? 

I I 10. Are vocabulary words defined in context and not obscuring learning goals? 

I I 1 1 . Is the layout of instructional materials interesting and appealing? 



BEST COPY AVAILABLE 
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C. Pedagogy 

I I 1 . Does almost half of the curriculum have students engage in active learning? 

I I 2. Do students construct knowledge through research, discussion, 
and application to gam conceptual understanding? 

I I 3. Are evaluation devices included and appropriate? (Highest points 
for authentic, performance-based assessment devices.) 

I I 4. Are instructional materials sensitive to social, economic, and cultural diversity? 

I I 5. Do lessons encourage students to develop awareness, knowledge, 
and strategies for responsible action? 

CZ! 6. Are group/cooperative learning strategies used? 

I I 7. Is intergenerational responsibility, linking today's actions 
with future consequences, implicit in instruction? 



D. Presentation 

□ 1 . Are instructions for the teacher clear and concise? 

I I 2. Are lesson objectives and outcomes clear and appropriate? 

I I 3. Are materials easily integrated into an established curriculum? 

im 4. Is background information for the teacher adequate and accurate? 

I I 5. Can the materials be adapted to vaned learning environments 

(large or small classes, of mixed levels, from rural or urban settings)? 

I I 6. Are consumable instructional materials of good quality, easily duplicated 
for student use, and in sufficient quantity to support the objectives? 

I I 7. Are equipment and materials listed and reasonably accessible? 

I I 8. Are a variety of instructional strategies, expanded learning 

environments, and resources suggested in the curriculum's design? 

I I 9. Is the time required to complete each lesson indicated? 

I I 10. Do the materials clearly list the subject discipline(s) integrated 
into each lesson? 



Used ond adapted with permission by thu Catitornia Department ol Education from the 
Fnviionmentat Education Comnendium lor Water Rcsourcei. The Cahtomia Department 
ol Education. The California Deoartment ol Water Resources: Sonoma State University. 
March 1992. 
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Learning to Look 



Objectives 

To introduce teachers to the power of 
pedagogy and how it will effect their 
evaluation. 



Materials 

Activities from Master 6 and 7 or 
two activities that cover the same 
topic. Optional: Copies of Master 
3, 4, and 5. 

Time 

40 minutes 



Preparation 

This activity' suggests teachers compare 
two EE activities tor students. One pair 
is provided, though you may substitute 
another from your own materials. If so, 
look for activities that are designed for 
the same age level, cover the same content, 
and have similar learning objectives, but 
use different teaching methods to reach 
the goals. For example, the activities on 
Masters 6 and 7 look at the value of farm- 
lands but use slighdy different approaches. 
"Down on the Farm" uses simulation and 
discussion. "Vanishing Farms" uses simu- 
lation and role play. 

•Other potential sets of activities are: 

"Water, Water Ever\'\\here" from Earth 
Matters, "How Wet is Our Planet" from 
Aquatic WILD, and "Eighty-six Gallons 
per Day" from Living Lightly on the Planet; 

"Bye Bye Birdie" from Eurtb Matters by 
ZPG and "Here Todav, Cnme Tomorrow" 
from Project WILD: ' 

"Watch on Wetlands" from Project Learn- 
ing Tree and "The Cireat Swamp Debate" 
in NatureScope's Wading into Wetlands. 



Outline 

Start a discussion among teachers about 
pedagogy and the design of instructional 
materials. Ask them to think about how 
these concepts affect the way we evaluate 
an activity or activity guide. Explain that 
a single activity rarely incorporates all the 
goals in the instructional guide. Duplicate 
the activities so each participant receives 
a pair. 

1 

Divide the participants into teams of four 
to five. Hand out copies of Masters 6 and 
7 to everyone. These activities have similar 
content but different objectives. Challenge 
the teams to critique both activities, noting 
the teaching methods, student outcomes, 
and ultimate objectives. You may wish to 
have the teams use one or more of the eval- 
uation tools from Masters 3, 4, or 5. 

2 

Lead a discussion about the difference 
in the two materials using the following 
questions: 

Would each activity' meet your 
teaching objectives? Why or why not? 

- What makes a good learning 
experience? 



What criteria would you use to 
evaluate each activity? 

What criteria ought you use for the 
pedagogical aspect ot the materials? 
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Master Q DowH Oil The Farm 



Objectives: 

Students will: 

• Measure and mark off an area of land which represents the amount of land required 
to feed them for one year 

♦ Investigate the conflict over private vs. public control of farmland resulting from 
agricultural zoning. 



Student information sheet provided: a long tape measure (at least 71 yards long) 
or, for a handy measuring device. use a yardstick, four small nails, 71 yards: of kite 
string, and a hammer. You will also need four stakes marked with brightly colored 
paint or material. 

Advance Preparation: 

To make the measuring device, pound two small nails into each end of a yardstick. 
Tie one end of the string to the nails at one end of the stick. 



Overview: 

It takes approximately 1.2 acres of land to feed one person each year This includes 
grazing land for dairy and beef cattle as well as cropland to grow grains, fruits, and vege- 
tables. Between 1967 and 1973 the amount of cropland and potential cropland convert- 
ed to urban uses was 875,000 acres per year This rate of cropland conversion is causing 
some concern in that we may be presenting future generations with a shortage of pro- 
ductive land. In an effort to stem the tide of cropland conversion, many communities are 
developing land use plans which include zoning laws to preserve farmland. The conflict 
that these laws create will be debated b> ..our students as they investigate this issue. 

1. Ask students how much land they think would be required to feed each of them for a 
one-year period. Round the 1.2 acre figure off to one acre and taKe your students out 
to measure an acre of land on your school grounds. An acre is 4,840 square yards 
(4,047 square meters). A square 69 yards, 20.5 inches on each side is an acre. (If the 
land you have to work with is not large enough, measure a one-half acre area. A half 
acre square is 49 yards, 7 inches on each side.) 

If you use the measuring device suggested above, have students wrap the string 
around the yardstick. Each loop around the length of the yardstick equals two yards 
of string. Have them wind the string until they have 69 yards, 20.5 inches and cut the 
string. Then use the string like a tape measure to mark off the four corners of the 
acre. Have some students hammer a stake into the ground at each corner 

2. Discuss the figures below that indicate what is happening to our individual share 
of cropland on a global basis. 



The decline of arable land per person is projected to decrease as the world population 
continues to expand. Most of the land suitable for farming is now in use. Only 1 1 per- 
cent of the world land is arable. The rest is too steep or cold or too dry to support 
agriculture. 



Materials: 



Year 

1650 
1975 
2135 



Cropland per person 

4 acres 
1 acres 

1/4 acres (projected) 
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Master Q 



3. Either read the information on the student sheet or distribute copies for students 
to read on their own. Discuss the issue of agricultural zoning and its infringement 
on individual property owner's right to sell their land as they choose. Review the 
discussion questions on the student's sheet and have students express their own 
points of view on the issue. 



Extensions: 

Write the following statement on the board: "Asphalt is the land's last crop." (The state- 
ment was made by M. Rupert Cutler, former United States Assistant Secretary of Agri- 
culture, in reference to the problem of vanishing farmland.) Have students use this as 
an introductory sentence m an essay expressing their views on the subject of farmland 
conversion. 

To help your students understand the special nature of topsoil, have them plant corn 
S2eds in a container of topsoil and in a container of subsoil. Subsoil is the layer of nutri- 
ent-poor soil to be found under humus-rich topsoil. It provides a poor growing medium 
and will help students to realize that not all land is suitable for growing food. 

Student Information 

How would you react to this situation if it were taking place in your community? 

Voices are rising along with tempers in the crowded room in City Hall. There is a public 
hearing being held to give citizens a chance to express their views about the proposed 
new zoning laws. Zoning laws set aside areas of land for specific uses. When a communi- 
ty has zoning laws, some areas are zoned for homes, others for industry, and other ar- 
eas are zoned for open space. In this community new zoning laws will restrict the uses 
of farmland. There is a hot debate taking place between people who are divided in their 
opinions. 

The proposed zoning would require that farmland be sold in tracts no smaller than 40 
acres and that the lands remain as open space or farmland. 

The community is near a growing city. The trend has been for farmland to be sold and 
divided into subdivisions and shopping malls. In many cases, it was more profitable for 
farmers to sell their land to developers than to continue to farm. Many farmers are an- 
gered over the proposed denial of their right to sell their own land for uses other than 
farming. They believe that people should have the right to do what they want with their 
own land. 

Other people in the room are concerned about the community's loss of farmland. They 
view the land as a valuable, irreplaceable resource. They argue that farmland should not 
be viewed as something to be bought or sold, but as a resource to be preserved for fu- 
ture generations They believe that preservation of farmland is a social obligation. This 
leads them to believe that the public, in this instance, has the right to exercise control 
over these valuable lands. The debate rages on.... What do you think? 



Reprinted with permission 
from Living Lightly on ttie 
Planet, through arrange- 
r"ent with Schlitz Audu- 
bon Center ol the National 
Audubon Society. 1111 E. 
Brown Deer Road. Milwau- 
kee, Wl 532! 7 



Discussion Questions 

• How would you react to the proposed zoning restrictions if you were a farmer? 
A housing developer? 

• Do you believe that the public, in the form of government, has the right to impose 
restrictions on individual landowners? State reasons for you beliefs. 

• Should land use decisions we make today consider the needs of generations which 
will follow us? Why or why not? 



46 



Master Q Vanishing Farms 



Objective: 

Students will gam an understanding of the pressures to develop open space and farms. 
Summary: 

Students will role-play farmer, crops, farm animals and the different types of interests 
that also desire land. 

Time: 

45 minutes 

Student Grouping: 

Whole class 

Materials: 

10-30 foot piece of string with ends tied together to make a circle. (Longer string for 
larger groups) 

Background Information: 

Every year three million acres of farmland are lost to some sort of development. If this 
continues, eventually there will not be enough land to grow food. Localities have begun to 
make ordinances and laws that facilitate maintaining agricultural land. The issue of pri- 
vate rights versus the common good is brought into play in these situations. 

Marin County, California. Agriculture Facts: 

The debate over land use is very real to Marin residents. For years there have been efforts 
to preserve county agricultural lands. Many people have been involved, particularly since 
the 1970's, in making it feasible for farmers to keep farming in spite of the skyrocketing 
prices of land in the area. The Williamson Act, passed in 1965 and amended in 1970, 
allows for reduced property taxes for agricultural lands within certain areas. MALT, the 
Marin Agricultural Land Trust, purchases agricultural easements, restricting future uses 
of property to agriculture and recreation. Without these types of provisions, much more 
development would be inevitable. 



Preparation: 

1. This is a very simple activity. You )ust need to be familiar with the exercise so it will 
flow smoothly in class. 

2. Acquire your string and decide where you will lay it out. 



Procedure: 

1. Place the string on the ground in a circle. 

2. Assign one student the role of "Farmer." 

3. The farmer lists the crops and livestock he would need in order to feed and clothe 
the local people. As the farmer names the things on his list (like corn, wheat, cows, 
sheep, fruit) a student is selected to step into the circle and represent the commodity 
named. 

4. One by one the rest of the students become people who also want to use the farm- 
land: for example, a road builder, someone developing a shopping center, someone 
digging for water, a school builder, a hotel owner These people move to the edge 
of the farmland, pick up the string and move it inward, making the circle smaller 
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5. Developers continue until the farm circle is too small to hold all the farmers livestock 
and crops. There has been too much development in the area and thus the farm has 
vanished. 

6. Discuss the activity w/ith the students. 
Questions for Discussion: 

• How/ does development pressure affect food production? 

• Would this affect prices? 

• What can all those developers do? 

• What can farmers do? 

• How/ does development and loss of farmland affect the families in the community? 
Extensions: 

• Have a farmer and developer come to the class and talk about their ideal plan 
for land use in the area. 

• Find some oid pictures of the school area (check w/ith a local historical society 
or local paper) and compare the open space then and now/. 

How does Marin Agricultural Land Trust work? 

Marin Agricultural Land Trust (MALT) seeks to permanently protect Marin's agricultural 
lands by purchasing agricultural easements from ranchers. 

An easement is a legal property right and can address many of the ways in which land 
can be used. An agricultural easement is one which allow the land to be used for agricul- 
tural purposes. MALT buys agricultural easements and then permanently assures that 
they are not sold off to allow subdivision of the land for development. 

Agricultural easements are always sold willingly by ranchers to allow them to get cash 
for farming operations, to make improvements, or meet other needs. The farmer selling 
the agricultural easement still owns the property and the land is guaranteed to stay in 
agricultural production. 

MALT was formed by a coalition of ranchers and environmentalists in 1980 and 
is considered a national model for agricultural land preservation. 



Used with permission and 
adapted from 'Vanishing 
Farms" bv Leeann 
Tourtillot't and Ellen Rilla. 
)992. In Food tor 

Growing Minds. University 
of California Cooperative 
Extension, Mann Agricui 
tural Literacy Project. 
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Activity 



0 What About 



Teacners ancuss the 
nevitabilitv of 0/as 
.md oractice aiscov- 
ffins It ;n materials. 



Objectives 

To enable teachers to detea bias 
in teaching materials and help them 
to develop strategies that overcome 
biases in EE instructional materials. 



Materials 

Handouts made from Masters 6 
and 7. 

Time 

30 minutes 



Oirtiine 

Begin by explaining that all materials have 
some bias toward the author s, producer's, 
or funder's viewpoints. Whether from in- 
dustry', textbook companies, government, 
or environmental organizations, most ma- 
terial includes a particular bias. 



Have teachers brainstorm the types of bias 
that they have deteaed in instructional 
materials. Some possible answers are: gen- 
der, age, culture, resource use vs. preserva- 
tion. 



Divide the participants into small groups 
and distribute Masters 6 and 7 to each. 
Ask the groups to list examples of where 
the author s bias comes through. 



Have groups share their insights. Note 
whether the groups came up with the same 
biases and examples. If there are differenc- 
es, ask the group to consider how their 
personal biases affected their perception of 
the materials. 



Ask how teachers might approach materi- 
als with bias. Add to their list any of the 
following: 

— Not use the material 

— Accept bias as is 

— Balance this with other materials 
of an opposite bias 

— Have students discover the bias 
as a learning experience 

— iMcdify materials to reduce bias 

Have each group discuss what they might 
do with these activities given the bias they 
found. Record all strategies on an over- 
head or newsprint. 

5 

(Optional) 

Ask participants to generate questions for 
an evaluation tool that would address their 
concerns about bias. 
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Resources 



Glossary 



- _ / 



"■•tocoK aermitior.'i 
•^av oe neiDtui m 
MO'jiain? j jditicn.i: 
■■'amoies 'or these 
:trms. Tnese aeiini- 
; ons .'e/er r.9 r-'i.? 

.■■.■rms aesc r;oea -/^ 



Evaluarion Tool/Instrument - The test, 
questionnaire, or checklist used to measure 
the performance of the instructional mate- 
rials to be evaluated. 

Formarive Evaluation - Evaluation done 
during development of instructional mate- 
rials for the purpose of rewriting and 
improving the quality of the final product. 

Measurement - Provides quantitative de- 
scriptions of the characteristics of a 
variable. 



Qualitative Data - Non-numerical infor- 
mation often obtained through open-end- 
ed questions, observations, or interviews. 

Quantitative Data - Numerical informa- 
tion from an evaluation. This information 
can be manipulated and may be reported 
as an average, rank, median, mean, or 
standard deviation. 

Reliability - For our uses this refers to the 
consistent nature of the evaluation tools' 
performance. Does the tool yield consis- 
tent results under similar circumstances? 
The degree of consistency determines the 
reliabilirv' of the tool. 



Standard Deviation - A statistical function 
that serves as measurement of the degree 
of agreement between responses for a spe- 
cific criterion on o given piece of material. 

Summative Evaluation - An evaluation 
done to examine the end result of a project 
or activity. Materials developers may con- 
duct a summative evaluation to measure 
whether their product does what it was 
intended. 

Validity - An estimate of the degree to 
which an evaluation tool measures what 
it is designed to measure. 

Variables - All the factors that could affect 
results of an evaluation project. For exam- 
ple, the evaluator may tire with a lengthy 
tool and therefore rush through the evalu- 
ation of the last few pieces of instructional 
material. 



i 



Significance - The odds that observed dif- 
ferences or similarities are due to the 
"real" influence of variables and not 
chance. 



I > V ' 
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Evaluated EE Activity Guides 

The Curricula and Compendia Project, 
Six Volumes: Energy Resources, Water 
Resources, Integrated Waste Management, 
Air Quality, Human Communities, and 
Natural Communities. California Depart- 
ment of Education, 1992-95. Sacramento, 
CA. These compendia contain the results 
of evaluations of K-12 environmental 
education curricula based on presentation, 
pedagogy, teacher usability, general con- 
tent, and subject-specific content. Each 
compendium includes a "report card," 
evaluator comments, a brief description, 
and sample pages. Available from: 
Bureau of Publications, Sales Unit 
California Department of Education 
P.O. Box 271 

Sacramento, CA 95812-0271 
© (916)445-1260 
Fax: 916 323-0823 

"Curriculum Materials for EE,'"D.L.Haur>', 
J.E. Heimlich, and M. Norton, edited by 
R.J. Wilke. In Environmental Education 
Teacher Resource Book, A Practical Guide 
for Environmental Education, 1993. 
Mil wood, NY: Kraus International Publi- 
cations. A selected, annotated listing of 
readily available materials for use in envi- 
ronmental education. Included are curricu- 
lum guides, supplementary materials, 
activities, nonprint media, modules, etc.), 
program frameworks, and resource materi- 
als. Available from: 

Kraus International Publications 

358 Saw Mill River Rd. 

Milkwood, NY 10546 
^ (914)762-2200 

Educating Young People About Water. 
A Guide to Goals and Resources with 
an Emphasis on Nonformal and School 
Enrichment Settings. Elaine Andrews, 
1992. Madison, Wl: University of Wiscon- 
sin. This guide was developed by the Water 
Curriculum Needs Assessment Project 
to help curriculum coordinators select and 



develop water quality training programs 
and curricula. Includes summaries of 63 
reviewed curricula. Available from: 

Elaine Andrews 

University of Wisconsin 

Environmental Resources Center 

216 Agriculture Hall 

1450 Linden Drive 

Madison, WI 53706 
■?) (608)262-6968 

Energy and the Environment: A Teacher's 
Resource Guide. Pacific Gas and Electric 
Company, 1991. San Francisco, CA. 
A catalog of energy education resources 
for elementary, secondary and higher 
education. Available from: 

Pacific Gas and Electric 

Energy and Environment Department 

77 Beale, 28th Floor 

San Francisco, CA 94106 
f© (415)973-4361 

"Evaluation of Selected Recycling 
Curricula: Educating the Green Citizen," 
S. Boerschig and R. DeYoung. \n Journal 
of Environmental Education, 24:17-22. 
1993. Evaluation of 14 solid waste curric- 
ula. Provides a ratings matrix of evaluated 
materials and recommendations for devel- 
opment of new materials. 



Evaluation Tools 

"Analyzing Educational Resources," Zita 
Unger, edited by John Fien. In Teaching for 
a Sustainable World: Environmental and 
Development Education Project for Teach- 
er Education. 1992. Brisbane, Australia: 
Griffith University Press. This workshop 
outline includes guidelines for group dis- 
cussions, overhead masters, readings, and 
an evaluation instrument to help partici- 
pants become engaged in evaluation issues, 
particularly bringing to light the range of 
ideologies and viewpoints that are embed- 
ded in resources. Available from: 
Dr. John Fien 

Faculty of Environmental Sciences 
Griffith Universit)' 
Nathan, Brisbane 
Australia 4111 
fax: 61-7-875-7459 
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"A Checklist for Reviewing Environmental 
Education Programs," Fred C. Niedermey- 
er. In Journal of Environmental Educa- 
tion, 23 (2):46-50. 1992. Enumerates, in a 
checklist form, the characteristics of well- 
developed, effective instructional programs 
to provide criteria for their review. 

Environmental Education Curriculum 
/«ye«foo'. J.R. Gardella, 1993. Highland 
Heights, KY: School of Education, North- 
ern Kentucky University. Includes two 
forms that use accepted goals of environ- 
mental education in a checklist format 
with a data summary device to identify the 
emphasis of the evaluated curricular mate- 
rials. Available from: 

North Kentucky University 
Education Department 
Highland Heights, KY 41099 
© (606)572-5624 

"Evaluating Sponsored Materials," Mary 
E. Haas. In How To Do It in the Social 
Studies Classroom Sen'es. 1985. 
Washington, D.C.: National Council for 
the Social Studies. Two evaluation forms 
are provided to carefully evaluate spon- 
sored materials and guest speakers. Avail- 
able from: 

The National Council 
for the Social Studies 

3501 Newark St., NW 

Washington. DC 20016 

(202)966-7840 



Additional Readings 

If you are interested in more information 
on evaluation or want to do a formal 
evaluation project we recommend these 
resources. 

"The CIPP Model for Program Evalua 
tion," D.L. Stufflebeam, edited by G.R 
Madaus, M. Scriven, & D.L. Stufflebeam. 
In Evaltdation Models. 1993. Boston: Klu- 
wer-Nijhoff. The CIPP approach is based 
on the view that the most important pur- 
pose of evaluation is not to prove but to 
improve. Evaluation is seen as a tool by 
which to help make programs work better 
for the people they are intended to serve. 
Available from: 

Kluwer Nijhoff Publications 

101 Phillip Drive 

Norwell, MA 02061 
•D (617)871-6600 

Criteria for Evaluating Instructional 
Products. ]zmts L. Moseley, 1987. ERIC 
Document Number: 281-870. This five- 
page checklist presents a rating scale ap- 
proach to evaluation instructional 
products and educational media. Fifteen 
categories of evaluation criteria are includ- 
ed. Available from: 
ERIC 

555 New Jersey Ave., NW 
Washington, DC 20208-5720 
(202)219-2088 
■t) (800)538-3742 

"Evaluating Environmental Education in 
the Context of a Junior High School State 
Studies Program," Dean Bennett. In 
Journal of Entnronmental Education, 
13(4): 13-18. 1982. Describes the Maine 
Studies Curriculum, a state-validated dem- 
onstration project taught as part of the 
regular school curriculum. Also describes 
the curriculum evaluation procedure and 
results. 
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"Evaluation of Environmental Education 
Programs at the Elementary and Secondary 
School Levels," Bernard J. Lucko, John E 
Disenger, and Robert E. Roth. In Journal 
of Environmental Education, 13 (4): 7-12. 
1982. Provides an overview of recent and 
current activity in environmental educa- 
tion, particularly as it relates to program 
evaluation in elementary and secondary 
schools. Also includes major steps in con- 
ducting an evaluation based on the goals 
of environmental education. 

"Evaluating Instructional Materials," Tim 
Wentling and William E. Piland. In Local 
Leader Guide DL Locally Directed Evalu- 
ation Handbook. Second Edition. 1982. 
Springfield, IL: Illinois Str-ite Board of Edu- 
cation. One of 12 guides that have been 
developed to facilitate evaluation by and 
for local education agency personnel in Illi- 
nois. Covers the evaluation of both print 
and audiovisual instructional materials. 
Available from: 

Curriculum Publications 
Clearing House 

Western Illinois University' 

Horrabin Hall #46 

Macomb, IL 61455 
® (800)322-3905 

"Evaluation of Natural Resource Educa- 
tion Materials: Implications for the Re- 
source Manager," Gerri Pomerantz. In 
Journal of Environmental Education, 21 
(2): 16-23. 1990. An analysis of elementa- 
ry school and natural resource lessons that 
focus on ecological principles, resource 
management issues, and analytical skill de- 
velopment. The conclusion is that very 
few of the lesson materials help to develop 
critical thinking skills and behaviors neces- 
sary for environmental action. 

Experimental and Quasi-Experimental 
Designs for Research. Donald T. Campbell 
and Julian C. Stanley, 1963. Chicago, IL: 
Rand McNally College Publishing Co. 
Examines the validity of 16 experimental 
designs against 12 common threats to val- 
id inference. Available from: 

Rand McNally College Publishing Co. 

444 N. Michigan Ave. 

Chicago, IL 60611 
'0 (312)321-1751 



Procedures for Developing an Environ- 
mental Education Curriculum. Harold 
Hungerford and Robert Ben Peyton, 
1986. EE Series 22. Paris, France: United 
Nations Educational, 

Scientific and Cultural Organization. ERIC 
document: ED 277 569. This publication 
is an update of a 1980 document and 
relates trends in environmental education 
to curriculum development. Available 
from: 

Unesco 

Environmental Education Unit 
7 place de Fontenoy 
75352 Paris 07 SP 
FRANCE 

A Scope and Sequence for Energy 
Education Grades K-ii. Sonoma State 
University, Environmental Studies and 
Planning Department, 1989. A conceptual 
matrix for evaluating criteria related to 
energy education. Available from: 

Sonoma State University 

Environmental Studies and Planning 

Department 

1801 East Copati Ave. 

Rohnert Park, CA 94928 
© (707)664-2306 

Standards for Evaluation of Instructional 
Materials with Respect to Social Content. 
California State Department of Education, 
1986. Overview of the California "Educa- 
tion Code" standards, which are intended 
to help dispel negative stereotypes by em- 
phasizing people in diverse, positive, and 
contributing roles. Available from: 

California State Department 
of Education 

P.O. Box 271 

Sacramento, CA 95802-0271 
(916)653-7026 
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A Qualitative Study of How One 
Teacloer Selects Instructional Materials. 
John Elliot, 1986. ERIC document: ED 
269-381 , This case study explores ways 
in which a primary grade teacher selects in- 
structional materials. Areas covered 
under the study include: the text as a set 
curriculum, reflections on the teacher's 
choices of instructional materials, the 
source of the teacher's instructional ideas, 
the teacher as "scrounger," instructional 
materials and parent involvement, and in- 
structional materials as outside experience. 
.Available from: 
ERIC 

555 New Jersey Ave., N\V 
Washington, DC 20208-5-20 
(202)219-2088 
0 (800)538-3742 

"What IS the Purpose ot Evaluation?" 
Cieorge T. O'Hearn, In journal 
of Environmental Education, 13 (4): 1-3. 
1982. Provides a broad, working definition 
of environmental education, suggesting 
that such broad definitions necessitate the 
identification of a few measurable objec- 
tives as "indicators" of an envir'inmental 
education program's impact. Discusses the 
purpose of and methods for evaluation. 



Endnotus 

1. Kirkpatnck, I)on;ild, "liv.iliKUion," Trjiiun^ 
and Development Handbook edited by 
Robert 1.. Ciraip, third edition. New York: 
McGraw-Hill. l^S"^. 

2. United N.uions Kducational, Scientific, and 
C'ultural Orijaiiization. 1-inal Report, 
biter^overmneiUdl (.ionference on Em m»i- 
incntjl Educatum, Uri;anii!ed by Unesco in 
Cloopcration with UNF.P, I bilisi, (icort;ia, 
USSR, 14-26 Oaober 14". P.uis; Unesco 
r.D/.\ 10/49, 

Ibid 

4. e.aliforni.i Dcpaitnieiit of rdiicatioii, Califor- 
nia Department ot Water Resources and 
Sonoma State Universitv. l.in irtmincntal 
I .dui.atuiii: (. .oi)if}fndiii»i for Water Kcsoiircr^. 
Sacramento. (. A. 1"42. 



^. Pacific Gas and Electric Company. Ener^' 
and the Environment: A Teachers Resource 
Guide. San Francisco, C!A, 1991. 

b, Tourtillott, l.eeann. Conserve and Renew, 
V.nergy Eiducation Activities for Grades 4 - 6, 
Rohnert Park, CA: Sonoma State University, 
1^90. 

~. Andrews, lilaine. Educating!, Youn^ People 
about Water: A Guide to Goals and 
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